Image Sensors—Capturing the Pixels Unlike traditional cameras that use film to capture and store an image, digital cameras use a solid-state device called an image sensor These fingernail-sized silicon chips contain millions of photosensitive diodes called photosites In the brief flickering instant that the shutter is open, each photosite records the intensity or brightness of the light that falls on it by accumulating a charge; the more light, the higher the charge The brightness recorded by each photosite is then stored as a set of numbers that can then be used to set the color and brightness of dots on the screen or ink on the printed page to reconstruct the image In this chapter, we’ll look closely at this process because it’s the foundation of everything that follows The Development of the CCD Based on a press release by Patrick Regan; Lucent Technologies, Murray Hill George Smith and Willard Boyle invented the charge-coupled device (CCD) at Bell Labs They were attempting to create a new kind of semiconductor memory for computers A secondary consideration was the need to develop solid-state cameras for use in video telephone service In the space of an hour on October , , they sketched out the CCD's basic structure, defined its principles of operation, and outlined applications including imaging as well as memory Willard Boyle (left) and George Smith (right) Courtesy of Lucent Technologies By , the Bell Labs researchers had built the CCD into the world's first solid-state video camera In , they demonstrated the first CCD camera with image quality sharp enough for broadcast television Today, CCD technology is pervasive not only in broadcasting but also in video applications that range from security monitoring to high-definition television, and from endoscopy to desktop videoconferencing Facsimile machines, copying machines, image scanners, digital still cameras, and bar code readers also have employed CCDs to turn patterns of light into useful information Since , when telescopes were first outfitted with solid-state cameras, CCDs have enabled astronomers to study objects thousands of times fainter than what the most sensitive photographic plates could capture, and to image in seconds what would have taken hours before Today all optical observatories, including the Hubble Space Telescope, rely on digital information systems built around "mosaics" of ultrasensitive CCD chips Researchers in other fields have put CCDs to work in applications as diverse as observing chemical reactions in the lab and studying the feeble light emitted by hot water gushing out of vents in the ocean floor CCD cameras also are used in satellite observation of the earth for environmental monitoring, surveying, and surveillance Image Sensors and Pixels Digital photographs are made up of hundreds of thousands or millions of tiny squares called picture elements, or just pixels Each of these pixels is captured by a single photosite on the image sensor when you take the photo Like the impressionists who painted wonderful scenes with small dabs of paint, your computer and printer can use these tiny pixels to display or print photographs To do so, the computer divides the screen or printed page into a grid of pixels, much like the image sensor is divided It then uses the values stored in the digital photograph to specify the brightness and color of each pixel in this grid—a form of painting by number Controlling, or addressing a grid of individual pixels in this way is called bit mapping and digital images are called bit-maps Here you see a reproduction of the famous painting "The Spirit of ‘ " done in jelly beans Think of each jelly bean as a pixel and it's easy to see how dots can form images Jelly Bean Spirit of ’ courtesy of Herman Goelitz Candy Company Inc Makers of Jelly Belly jelly beans The makeup of a pixel varies depending on whether it's in the camera, on the screen, or on a printout On an image sensor, each photosite captures the brightness of a single pixel The layout of the photosites can take the form of a grid or honeycomb depending on who designed it A typical image sensor has square photosites arranged in rows and columns The Super CCD from Fuji uses octagonal pixels arranged in a honeycomb pattern Image size The quality of a digital image, whether printed or displayed on a screen, depends in part on the number of pixels used to create the image (sometimes referred to as resolution) The maximum number that you can capture depends on how many photo sites there are on the image sensor used to capture the image (However, some cameras add additional pixels to artificially inflate the size of the image You can do the same thing in an image-editing program In most cases this upsizing only makes the image larger without making it better ) Image Sizes—Optical and Interpolated Beware of claims about image sizes (often referred to as resolution) for cameras and scanners because there are two kinds; optical and interpolated The optical resolution of a camera or scanner is an absolute number because an image sensor's photosites are physical devices that can be counted To improve resolution in certain limited respects, the resolution can be increased using software This process, called interpolated resolution, adds pixels to the image To do so, software evaluates those pixels surrounding each new pixel to determine what its colors should be For example, if all of the pixels around a newly inserted pixel are red, the new pixel will be made red What's important to keep in mind is that interpolated resolution doesn't add any new information to the image—it just adds pixels and makes the file larger This same thing can be done in a photo editing program such as Photoshop by resizing the image Beware of companies that promote or emphasize their device's interpolated (or enhanced) resolution You're getting less than you think you are Always check for the device's optical resolution If this isn't provided, flee the product—you're dealing with marketing people who don't have your best interests at heart More pixels add detail and sharpen edges If you enlarge any digital image enough, the pixels will begin to show-an effect called pixelization This is not unlike traditional silver-based prints where grain begins to show when prints are enlarged past a certain point The more pixels there are in an image, the more it can be enlarged before pixelization occurs The photo of the face (right) looks normal, but when the eye is enlarged too much (left) the pixels begin to show Each pixel is a small square made up of a single color This table lists some standards of comparison The numbers from various sources differ One great thing about the Web is that you can talk back to an author and correct him Click here to send a message setting me straight Element Resolution Total Pixels Color TV (NTSC) x , Human eye , x , million -mm slide The "Economist" magazine says it has million or more CMOS Imaging News says to million depending on the film Another source says about million pixels Robert Caspe at SoundVision states that color negative film has pixels per inch while color positive film has pixels per inch Kodak Disc camera film million pixels—each about inch in diameter The size of a photograph is specified in one of two ways-by its dimensions in pixels or by the total number of pixels it contains For example, the same image can be said to have x pixels (where "x" is pronounced "by" as in " by "), or to contain -million pixels ( multiplied by ) This digital image of a Monarch butterfly chrysalis is pixels wide and pixels tall It's said to be x Camera Resolutions As you have seen, image sensors contain a grid of photosites—each representing one pixel in the final image The sensor's resolution is determined by how many photosites there are on its surface This resolution is usually specified in one of two ways—by the sensor's dimension in pixels or by its total number of pixels For example, the same camera may specify its resolution as x pixels (where "x" is pronounced "by" as in " by "), or -thousand pixels ( multiplied by ) Very high end cameras often refer to file sizes instead of resolution For example, someone may say a camera creates -Megabyte files This is just a form of shorthand Low-end cameras currently have resolutions around x pixels, although this number constantly improves Better cameras, those with million or more pixels are called megapixel cameras and those with over -million are called multi-megapixel cameras Even the most expensive professional digital cameras give you only about -million pixels As you might expect, all other things being equal, costs rise with the camera's resolution Size isn't everything! The large an image's size in pixels, the larger the image file needed to store it For this reason, some cameras allow you to specify more than one size when you take a picture Although you are likely to get better results with a larger image, it isn't always needed—especially when the image is going to be displayed on the Web or printed very small In these cases smaller images will suffice and because they have smaller file sizes, you'll be able to squeeze more into the camera's memory Resolution of Digital Devices Although more photosites often means better images, adding more isn't easy and creates other problems For example: It adds significantly more photosites to the chip so the chip must be larger and each photosite smaller Larger chips with more photosites increase difficulties (and costs) of manufacturing Smaller photosites must be more sensitive to capture the same amount of light More photosites create larger image files, creating storage problems Monitor Resolutions The resolution of a display monitor is almost always given as a pair of numbers that indicate the screen's width and height in pixels For example, a monitor may be specified as being x , x , x , and so on The first number in the pair is the number of pixels across the screen The second number is the number of rows of pixels down the screen This is a x display That means there are pixels on each row and there are rows Images displayed on the monitor are very low-resolution As you can see from the table below, the actual number of pixels per inch depends on both the resolution and the size of the monitor Generally, images that are to be displayed on the screen are converted to pixels per inch (ppi), a resolution held over from an early era in Apple's history (The red numbers in the table are the pixels per inch for each combination of screen size and resolution ) As you can see from the table, this isn't an exact number for any resolution on any screen, but it tends to be a good compromise If an image is pixels wide, the pixels per inch are different on a -inch wide monitor that on a -inch The same number of pixels have to be spread over a larger screen so the pixels per inch falls Resolution Monitor Size x x x Printer and Scanner Resolutions Printer and scanner resolutions are usually specified by the number of dots per inch (dpi) that they print or scan (Generally pixels per inch refer to the image and display screen and dots per inch refer to the printer and printed image Sometimes I think terminology shifts like this are done just to confuse us In this book we use them interchangeably) For comparison purposes, monitors use an average of ppi to display text and images, ink-jet printers range up to dpi or so, and commercial typesetting machines range between , and , dpi Image Sensors Just as in a traditional camera, light enters a digital camera through a lens controlled by a shutter Digital cameras have one of three types of electronic shutters that control the exposure: Electronically shuttered sensors use the image sensor itself to set the exposure time A timing circuit tells it when to start and stop the exposure Electromechanical shutters are mechanical devices that are controlled electronically Electro-optical shutters are electronically driven devices in front of the image sensor which change the optical path transmittence From Light Beams to Images When the shutter opens, rather than exposing film, the digital camera collects light on an image sensor—a solid state electronic device As you've seen, the image sensor contains a grid of tiny photosites As the lens focuses the scene on the sensor, some photosites record highlights, some shadows, and others record all of the levels of brightness in between Image sensors are often tiny devices Here you can see how much smaller the common / " and / " sensors are compared to a mm slide or negative Each site converts the light falling on it into an electrical charge The brighter the light, the higher the charge When the shutter closes and the exposure is complete, the sensor "remembers" the pattern it recorded The various levels of charge are then converted to digital numbers that can be used to recreate the image Image sensors contain a grid of photosites that convert light shining on them to electrical charges These charges can then be measured and converted into digital numbers that indicate how much light hit each site Courtesy of VISION These two illustrations show how image sensors capture images When an image is focused through the camera (or scanner) lens, it falls on the image sensor Varying amounts of light hit each photosite and knock loose electrons that are then captured and stored The number of electrons knocked loose from any photosite is directly proportional to the amount of light hitting it When the exposure is completed, the sensor is like a checkerboard, with different numbers of checkers (electrons) piled on each square (photosite) When the image is read off the sensor, the stored electrons are converted to a series of analog charges which are then converted to digital values by an Analog-to-Digital (A to D) converter Interlaced vs Progressive Scan Once the sensor has captured an image, it must be read, converted to digital, and then stored The charges stored on the sensor are not read all at once but a row at a time There are two ways to do this—using interlaced or progressive scans On an interlaced scan sensor, the image is first processed by the odd lines, and then by the even lines These kinds of sensors are frequently used in video cameras because television broadcasts are interlaced On a progressive scan sensor, the rows are processed one after another in sequence On an interlaced scan sensor, the image is first read off every other row, top to bottom The image is then filled in as each alternate row is read Image Sensors and Colors When photography was first invented, it could only record black & white images The search for color was a long and arduous process, and a lot of hand coloring went on in the interim (causing one author to comment "so you have to know how to paint after all!") One major breakthrough was James Clerk Maxwell's discovery that color photographs could be formed using red, blue, and green filters He had the photographer, Thomas Sutton, photograph a tartan ribbon three times, each time with a different one of the color filters over the lens The three images were developed and then projected onto a screen with three different projectors, each equipped with the same color filter used to take its image When brought into register, the three images formed a full color image Over a century later, image sensors work much the same way Additive Colors Colors in a photographic image are usually based on the three primary colors red, green, and blue (RGB) This is called the additive color system because when the three colors are combined in equal quantities, they form white This system is used whenever light is projected to form colors as it is on the display monitor (or in your eye) The first commercially successful use of this system to capture color images was invented by the Lumerie brothers in and became know as the Autochrome process They dyed grains of starch red, green, and blue and used them to create color images on glass plates RGB uses additive colors When all three are mixed in equal amounts they form white When red and green overlap they form yellow, and so on For more on color, visit The ColorCube Web site On the monitor, each pixel is formed from a group of three dots, one each for red, green, and blue On the screen, each pixel is a single color formed by mixing triads of red, green, and blue dots or LCDs Subtractive Colors Although most cameras use the additive RGB color system, a few high-end cameras and all printers use the CMYK system This system, called subtractive colors, uses the three primary colors Cyan, Magenta, and Yellow (hence the CMY in the name—the K stands for an extra black) When these three colors are combined in equal quantities, the result is a reflected black because all of the colors are subtracted The CMYK system is widely used in the printing industry, but if you plan on displaying CMYK images on the screen, they have to be converted to RGB and you lose some color accuracy in the conversion When you combine cyan, magenta, and yellow inks or pigments, you create subtractive colors On a printout, each pixel is formed from smaller dots of cyan, magenta, yellow, and black ink Where these dots overlap, various colors are formed This brief animation zooms in on a part of an inkjet print to show you increasing levels of detail Courtesy of Trevor Anderson It's All Black and White After All Image sensors record only the gray scale—a series of increasingly darker tones ranging from pure white to pure black Basically, they only capture brightness The gray scale contains a range of tones from pure white to pure black How then, do sensors capture colors when all they can do is record grays? The trick is to use red, green, and blue filters to separate out the red, green and blue components of the light reflected by an object (Likewise, the filters in a CMYK sensor will be either cyan, magenta, or yellow ) There are a number of ways to do this, including the following: Three separate image sensors can be used, each with its own filter This way each image sensor captures the image in a single color Three separate exposures can be made, changing the filter for each one In this way, the three colors are "painted" onto the sensor, one at a time Filters can be placed over individual photosites so each can capture only one of the three colors In this way, one-third of the photo is captured in red light, one-third in blue, and one-third in green Each pixel on the image sensor has red, green, and blue filters intermingled across the photosites in patterns designed to yield sharper images and truer colors The patterns vary from company to company but the most popular is the Bayer mosaic pattern shown here behind the image sensor Courtesy of IBM From Black and White to Color When three separate exposures are made through different filters, each pixel on the sensor records each color in the image and the three files are merged to form the full-color image However, when three separate sensors are used, or when small filters are placed directly over individual photosites on the sensor, the optical resolution of the sensor is reduced by one-third This is because each of the available photosites records only one of the three colors For example, on some sensors with million photosites, -thousand have red filters, -thousand have blue, and -thousand have green Does this mean the resolution is still million, or is it now -thousand? Or -thousand? Let's see Each site stores its captured color (as seen through the filter) as an -, -, or -bit value To create a -, -, or -bit full-color image, interpolation is used This form of interpolation uses the colors of neighboring pixels to calculate the two colors a photosite didn't record By combining these two interpolated colors with the color measured by the site directly, the original color of every pixel is calculated ("I'm bright red and the green and blue pixels around me are also bright so that must mean I'm really a white pixel ") This step is computer intensive since comparisons with as many as eight neighboring pixels is required to perform this process properly; it also results in increased data per image so files get larger Here the full-color of the center green pixel is about to be interpolated from the colors of the eight surrounding pixels HP has introduced a process called demosaicing that interpolates colors using a much wider range of adjacent pixels Courtesy of HP Color Channels Each of the colors in an image can be controlled independently and is called a color channel If a channel of -bit color is used for each color in a pixel—red, green, and blue—the three channels can be combined to give -bit color When an image is open in Photoshop, a dialog box shows the red, green, and blue channels so you can select the one you want to work on The top image in the dialog box is the combined -bit RGB Color Aliasing When interpolation is used, there has to be enough information in surrounding pixels to contribute color information This isn't always the case Low-resolution image sensors have a problem called color aliasing that occurs when a spot of light in the original scene is only big enough to be read by one or two pixels Surrounding pixels don't contain any accurate color information about the pixel so the color of that spot may show up as a dot of color disconnected from the surrounding image Another form of color aliasing shows up as out of place color fringes surrounding otherwise sharply defined objects Area Array and Linear Sensors Hand a group of camera or scanner designers a theory and a box of components and you'll see fireworks They will explore every possible combination to see which works best The market determines the eventual winners in this "throw them against the wall and see what sticks" approach At the moment, designers have two types of components to play with: area array and linear sensors Area-array Sensors Most cameras use area-array sensors with photosites arranged in a grid because they can cover the entire image area and capture an entire image all at once Area array image sensors have their photosites (pixels) arranged in a grid so they can instantly capture a full image Courtesy of VISION These area array sensors can be incorporated into a camera in a variety of ways One-chip, one-shot cameras use different color filters over each photosite to capture all three colors with a single exposure This is the most common form of image sensor used in consumer-level digital cameras One chip, three shot cameras take three separate exposures: one each for red, green, and blue A different colored filter is placed in front of the image sensor for each of the colors These cameras cannot photograph moving objects in color (although they can in black & white) and are usually used for studio photography Two-chip cameras capture chromonance using one sensor (usually equipped with filters for red light and blue light) and luminance with a second sensor (usually the one capturing green light) Two-chip cameras require less interpolation to render true colors Three-chip cameras, such as one from MegaVision, use three full frame image sensors; each coated with a filter to make it red-, green- or blue-sensitive A beam splitter inside the camera divides incoming images into three copies; one aimed at each of the sensors This design delivers high-resolution images with excellent color rendering However, three-chip cameras tend to be both costly and bulky Linear Sensors Scanners, and a few professional cameras, use image sensors with photosites arranged in either one row or three Because these sensors don't cover the entire image area, the image must be scanned across the sensor as it builds up the image from the captured rows of pixels Cameras with these sensors are useful only for motionless subjects and studio photography However, these sensors are widely used in scanners As a linear sensor scans an image a line at a time it gradually builds up a full image Linear image sensors put a different color filter over the device for three separate exposures—one each to capture red, blue or green Tri-linear sensors use three rows of photosites—each with a red, green, or blue filter Since each pixel has it's own sensor, colors are captured very accurately in a single exposure CCD And CMOS Image Sensors Until recently, CCDs were the only image sensors used in digital cameras They have been well developed through their use in astronomical telescopes, scanners, and video camcorders However, there is a new challenger on the horizon, the CMOS image sensor that promises to eventually become the image sensor of choice in a large segment of the market Image sensors are formed on silicon wafers and then cut apart Courtesy of IBM CCD Image Sensors Charge-coupled devices (CCDs) capture light on the small photosites on their surface and get their name from the way that charge is read after an exposure To begin, the charges on the first row are transferred to a read out register From there, the signals are then fed to an amplifier and then on to an analog-to-digital converter Once the row has been read, its charges on the read-out register row are deleted, the next row enter the read-out register, and all of the rows above march down one row The charges on each row are "coupled" to those on the row above so when one moves down, the next moves down to fill its old space In this way, each row can be read—one row at a time The CCD shifts one whole row at a time into the readout register The readout register then shifts one pixel at a time to the output amplifier It is technically feasible but not economic to use the CCD manufacturing process to integrate other camera functions, such as the clock drivers, timing logic, and signal processing on the same chip as the photosites These are normally put on separate chips so CCD cameras contain several chips, often as many as , and not fewer than CMOS Image Sensors Image sensors are manufactured in wafer foundries or fabs Here the tiny circuits and devices are etched onto silicon chips The biggest problem with CCDs is that there isn't enough economy of scale They are created in foundries using specialized and expensive processes that can only be used to make CCDs Meanwhile, more and larger foundries across the street are using a different process called Complementary Metal Oxide Semiconductor (CMOS) to make millions of chips for computer processors and memory This is by far the most common and highest yielding process in the world The latest CMOS processors, such as the Pentium III, contain almost million active elements Using this same process and the same equipment to manufacturer CMOS image sensors cuts costs dramatically because the fixed costs of the plant are spread over a much larger number of devices (CMOS refers to how a sensor is manufactured, and not to a specific sensor technology ) As a result of this economy of scale, the cost of fabricating a CMOS wafer is one-third the cost of fabricating a similar wafer using a specialized CCD process The CCD imaging elements used in most digital cameras are costly and consume high levels of energy Increasing the size of these imaging elements to include more pixels would require much larger power supplies as well as make them even more expensive In developing the EOS D , Canon concentrated on using a CMOS sensor instead of a CCD The CMOS technology has attracted attention for its lower power requirements, as well as its ability to integrate with image-processing circuits However, several engineering obstacles remained–including problems with the precision of internal transistors that led to variable image precision–and few products were developed using this technology However, Canon has now overcome these obstacles, and has succeeded in developing a large CMOS sensor Courtesy of Canon Here are some things you might like to know about CMOS image sensors: CMOS image quality is now matching CCD quality in the low- and mid-range, leaving only the high-end image sensors still unchallenged CMOS image sensors can incorporate other circuits on the same chip, eliminating the many separate chips required for a CCD This also allows additional on-chip features to be added at little extra cost These features include anti-jitter (image stabilization) and image compression Not only does this make the camera smaller, lighter, and cheaper; it also requires less power so batteries last longer CMOS image sensors can switch modes on the fly between still photography and video However, video generates huge files so initially these cameras will have to be tethered to the mothership (the PC) when used in this mode for all but a few seconds of video However, this mode works well for video conferencing although the cameras can't capture the frames a second needed for full-motion video While CMOS sensors excel in the capture of outdoor pictures on sunny days, they suffer in low light conditions Their sensitivity to light is decreased because part of each photosite is covered with circuitry that filters out noise and performs other functions The percentage of a pixel devoted to collecting light is called the pixel’s fill factor CCDs have a % fill factor but CMOS cameras have much less The lower the fill factor, the less sensitive the sensor is and the longer exposure times must be Too low a fill factor makes indoor photography without a flash virtually impossible To compensate for lower fill-factors, micro-lenses can be added to each pixel to gather light from the insensitive portions of the pixel and "focus" it down to the photosite In addition, the circuitry can be reduced so it doesn't cover as large an area Fill factor refers to the percentage of a photosite that is sensitive to light If circuits cover % of each photosite, the sensor is said to have a fill factor of % The higher the fill factor, the more sensitive the sensor CMOS sensors have a higher noise level than CCDs so the processing time between pictures is higher as these sensors use digital signal processing (DSP) to reduce or eliminate the noise The DSP is one early camera (the Svmini), executes , , instructions per picture 